Thanks @yufenglee for PR! Do you have before and after perf numbers for the repro in this issue #19924 ?

Thanks @yufenglee for PR! Do you have before and after perf numbers for the repro in this issue #19924 ?

on my local box, it take ~1.5ms before and ~0.5ms after.

May need take a look at cost model approach to see why cost model cannot work properly since it is a fundamental for CPU EP. Maybe try use correct cost (like adding concat kv cost etc) to see whether it could resolve the issue as well.

Please run benchmark for comparison of before/after:

• BERT base, batch size 1/2/4, different sequence lengths 16/64/128/256/512 on different interop_num_threads=1, 2, 4, 8
• GPT2 for decoding, batch size 1/2/4, with past seq len 16/64/128/256/512 on different interop_num_threads=1, 2, 4, 8
  Just in case, in some situations, cost model is better.

Description

The CostModel of threading doesn't not work for the attention especially for decoding case. It leads to less thread to compute the attention. Each batch*num_of_head is sufficient to serve as one unit. Change to use the TrySimpleParallelFor

Motivation and Context

#19924

Would change loop_len to batch_size * num_heads_ * sequence_length solve the issue?

The cost computation of ComputeVxAttentionScore is wrong. It should be sequence_length * v_head_size * total_sequence_length instead of sequence_length * v_head_size * sequence_length.

Also fine-tuned the cost computation for data load and store.